Compensating for a sensor deficiency in a heterogeneous sensor array

ABSTRACT

Apparatuses, methods and storage medium associated with compensating for a sensor deficiency in a heterogeneous sensor array are disclosed herein. In embodiments, an apparatus may include a compute device to aggregate perception data from individual perception pipelines, each of which is associated with respective one of different types of sensors of a heterogeneous sensor set, to identify a characteristic associated with a space to be monitored by the heterogeneous sensor set; detect a sensor deficiency associated with a first sensor of the sensors; and in response to a detection of the sensor deficiency, derive next perception data for more than one of the individual perception pipelines from sensor data originating from at least one second sensor of the sensors. Other embodiments may be disclosed or claimed.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. App. No. 16/614,741filed Nov. 18, 2019, which is a national phase entry under 35 U.S.C. §371 of Int’l App. No. PCT/US2017/042849 filed Jul. 19, 2017, whichdesignated, among the various States, the United States of America, thecontents of each of which are hereby incorporated by reference in theirentireties and for all purposes.

TECHNICAL FIELD

The present disclosure relates to the field of autonomous orsemi-autonomous apparatuses, and more specifically relates tocompensating for a sensor deficiency in a heterogeneous sensor array.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart by inclusion in this section.

Autonomous or semi-autonomous apparatuses, such as self-drivingvehicles, unmanned aerial vehicles (UAV, also referred to as drones), orrobots, may rely on a multimodal set of sensors to perceive, map, andtrack the surroundings. The sensors may include several types such aslong-range radar, mid-range radar front, night vision camera, videocamera, reverse camera, ultrasound, mid-range radar back, or the like.Each type of sensor may have its own advantages and deficiencies.

Algorithms used by autonomous or semi-autonomous apparatuses (such asroad detection algorithms, lane detection algorithms, traffic lightdetection algorithms, etc.) may be depend on a specific type of sensorof the set of sensors properly functioning. As a result of thisdependency, if one of the sensors malfunctions (e.g., stops generatingsensor data), capabilities of the autonomous or semi-autonomousapparatuses related to these algorithms may be completely lost.Autonomous or semi-autonomous apparatuses may be programmed to stop orland safely in such a condition.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be readily understood by the following detaileddescription in conjunction with the accompanying drawings. To facilitatethis description, like reference numerals designate like structuralelements. Embodiments are illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1 illustrates an example system equipped with technology forcompensating for a sensor deficiency in a heterogeneous sensor array,according to various embodiments.

FIG. 2 illustrates another example system equipped with technology forcompensating for a sensor deficiency of sensors of an autonomous vehicleor a vehicle with an automated driver warning system, according tovarious embodiments.

FIG. 3 is a flow chart showing a process of compensating for a sensordeficiency, according to various embodiments.

FIG. 4 illustrates a sensor translation model, in some embodiments.

FIG. 5 illustrates an example compute device that may employ theapparatuses and/or methods described herein, according to variousembodiments.

FIG. 6 illustrates monitored zones of a heterogeneous sensor array of amotor vehicle, according to various embodiments.

DETAILED DESCRIPTION

Apparatuses, methods and storage medium associated with compensating fora sensor deficiency in a heterogeneous sensor array are disclosedherein. In embodiments, an apparatus to navigate along a trajectory mayinclude a heterogeneous sensor array to monitor navigation along saidtrajectory, the heterogeneous sensor array including a plurality ofdifferent types of sensors; and a perception engine to: aggregateperception data from individual perception pipelines, each of which isassociated with a respective one of the sensors, to identify acharacteristic associated with the trajectory; detect a sensordeficiency associated with a first sensor of the sensors; and inresponse to a detection of the sensor deficiency, derive next perceptiondata for more than one of the individual perception pipelines based onsensor data originating from at least one second sensor of the sensors.

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof wherein like numeralsdesignate like parts throughout, and in which is shown by way ofillustration embodiments that may be practiced. It is to be understoodthat other embodiments may be utilized and structural or logical changesmay be made without departing from the scope of the present disclosure.Therefore, the following detailed description is not to be taken in alimiting sense, and the scope of embodiments is defined by the appendedclaims and their equivalents.

Aspects of the disclosure are disclosed in the accompanying description.Alternate embodiments of the present disclosure and their equivalentsmay be devised without parting from the spirit or scope of the presentdisclosure. It should be noted that like elements disclosed below areindicated by like reference numbers in the drawings.

Various operations may be described as multiple discrete actions oroperations in turn, in a manner that is most helpful in understandingthe claimed subject matter. However, the order of description should notbe construed as to imply that these operations are necessarily orderdependent. In particular, these operations may not be performed in theorder of presentation. Operations described may be performed in adifferent order than the described embodiment. Various additionaloperations may be performed and/or described operations may be omittedin additional embodiments.

For the purposes of the present disclosure, the phrase “A and/or B”means (A), (B), or (A and B). For the purposes of the presentdisclosure, the phrase “A, B, and/or C” means (A), (B), (C), (A and B),(A and C), (B and C), or (A, B and C).

The description may use the phrases “in an embodiment,” or “inembodiments,” which may each refer to one or more of the same ordifferent embodiments. Furthermore, the terms “comprising,” “including,”“having,” and the like, as used with respect to embodiments of thepresent disclosure, are synonymous.

As used herein, the term “circuitry” may refer to, be part of, orinclude an Application Specific Integrated Circuit (ASIC), an electroniccircuit, a processor (shared, dedicated, or group) and/or memory(shared, dedicated, or group) that execute one or more software orfirmware programs, a combinational logic circuit, and/or other suitablecomponents that provide the described functionality.

In devices utilizing a multimodal set of sensors, the device maycompensate for a sensor deficiency (e.g., sensor inactivity, a sensormalfunction, a sensor failure, a sensor reset, a security eventassociated with at least one of the sensors) in one modality byexploiting the information obtained from another modality. The devicemay be a component of a vehicle (such as a wheeled road or off-roadmotor vehicle or any other type of vehicle such as a vehicle to operateon rails, an aircraft, a watercraft, a robot, or the like, orcombinations thereof) or a non-vehicle such as a stationary surveillancesystem that uses more than one sensor type.

In some applications, each sensor of a heterogeneous sensor array maymonitor a different physical space. For instance, in a heterogeneoussensor array in a motor vehicle embodiment, the sensors may monitordifferent spaces around the motor vehicle such as a particular zone infront of the motor vehicle or a particular zone behind the motorvehicle. Different types of sensors that may monitor respective zones infront of the motor vehicle may include: ultrasound, video camera (dayand/or night vision), mid-range radar, and long range radar, or thelike. Different types of sensors that may monitor respective zonesbehind the motor vehicle may include: ultrasound, reverse camera, or thelike. The monitored zone of one sensor may overlap (e.g., partiallyoverlap) by a monitored zone of another sensor. FIG. 6 illustratesmonitored zones 601-607 around a motor vehicle (e.g., long range radarzone 601, mid-range radar front zone 602, night vision camera zone 603,video camera zone 604, ultrasound zone 605, mid-range radar back zone606, and reverse camera zone 607), some of which are overlapping (e.g.,partially overlapping).

In some embodiments, the device may use a sensor translation model toconvert sensor data obtained from one modality into another modality.Upon sensor deficiency, the sensor translation model may be activatedand/or provide the same input as the missing/defective sensors based onthe available working sensors. In embodiments in which the sensortranslation model is utilized in an autonomous vehicle (e.g., anautonomous motor vehicle), this conversion may allow the autonomousvehicle to operate safely in the event of a sensor malfunction withoutthe need to modify the logic that governs the operational behavior ofthe autonomous vehicle. Also, the autonomous vehicle may not need to beequipped with a redundant sensor of the same type to operate safely inthe case of a sensor malfunction (although a motor vehicle that isequipped with a duplicate sensor of the same type may of course utilizethis conversion to operate safely in the event of a sensor deficiencyaffecting both of a sensor and its redundancy).

Some embodiments may utilize machine learning for minimizing the impactof the loss of an active sensor by compensating the lack of criticalinput data with reliable artificially generated data. Even though thequality of the “virtual” sensor data may not be as accurate as the lostsensor, the vehicle may still be able to maintain a complete perceptionpipeline and keep its operational capabilities. Before furtherdescribing the compensation technology, it should be noted, for ease ofunderstanding, examples may be described for particular autonomous orsemi-autonomous applications, self-driving vehicles, UAV, robot, and soforth. Presentation of applications in a particular autonomous orsemi-autonomous application is not meant to be limiting. In general, theexamples while described in context of one application, is nonethelessillustrative across the various autonomous or semi-autonomousapplications.

FIG. 1 illustrates an example system 100 equipped with technology forcompensating for a sensor deficiency in a heterogeneous sensor array 1,according to various embodiments. The system 100 includes aheterogeneous sensor array 1 and a perception engine 15 to generatesensor data 19 (e.g., modeled sensor data) based on a sensor arrayoutput 11.

The heterogeneous sensor array 1 may include any number of differenttypes of sensors 5A-N. The sensors 5A-N may generate raw sensor outputs6A-N, respectively. Each one of raw sensor outputs 6A-N may provideinformation about a different physical space monitored by the sensors5A-N.

When all sensors 5A-N are operating normally, sensor array output 11 mayinclude all of the raw sensor outputs 6A-N (and/or may include processedinformation based on all of the raw sensor outputs 6A-N). However, inthe case of a sensor deficiency, sensor array output 11 may include onlysome of the raw sensor outputs 6A-N (and/or may include processedinformation based on only some of the raw sensor outputs 6A-N).

Perception engine 15 may detect a sensor deficiency of at least one ofthe sensors 5A-N. In embodiments in which perception engine 15aggregates data from individual perception pipelines associated with rawsensor outputs 6A-N, respectively, perception engine 15 may detect thesensor deficiency by identifying a gap in one of the individualperception pipelines. Alternatively, or in addition to identifying asensor deficiency based on a gap in a perception pipeline, perceptionengine 15 may receive control signals generated by one of the sensors5A-N or another component, which may indicate a sensor deficiency (e.g.,a rise or fall of a corresponding one of the control signals).

Following detection of a sensor deficiency, perception engine 15 maygenerate sensor data 19 (e.g., modeled sensor data) corresponding to thesensor associated with the sensor deficiency, from the remaining rawsensor outputs 6A-N. In other embodiments, the perception engine 15 mayconstantly generate modeled sensor data and may select a portion of theconstantly generated modeled sensor data responsive to detection of thesensor deficiency. The perception engine 15 may identify the sensor data19 responsive to the selection.

The sensor data 19 may be used to compensate for the sensor deficiency.For instance, in a motor vehicle including a LIDAR (light detection andranging) sensor, a computer vision sensor (e.g., monocular/stereo/depthcameras), and a radar sensor, the device may compensate for a sensordeficiency of one of the LIDAR sensor, the camera, or the radar usingsensor data generated by one or more of the others of the LIDAR sensor,the camera, or the radar.

In some embodiments, the perception engine 15 may feed sensor data 19(which may be generated based on content from a subset of the perceptionpipelines) back into one or more of the individual perception pipelinesthat is/are associated with the sensor deficiency(s) to fill aperception pipeline gap or prevent a potential perception pipeline gap.The content of all the individual perception pipelines may then beaggregated to identify a characteristic associated with a space to bemonitored by the sensor array 1. The characteristic may be the presenceof an object if the heterogeneous sensor array 1 is used in asurveillance system, or the characteristic may be a characteristic of atrajectory (such as whether an obstacle is located along the trajectory)if the heterogeneous sensor array 1 is used in a vehicle, e.g., anautonomous vehicle or a vehicle with an automated driver warning system.

In some embodiments, an automated warning system to display data to adriver and/or a vehicle component to change a trajectory of the vehicle(an automatic braking and/or steering component) may receive the sensordata 19 or data derived from the sensor data 19 as an input. Algorithmsused by this vehicle component (such as road detection algorithms, lanedetection algorithms, traffic light detection algorithms, etc.) maycontinue to function based on the sensor data 19, and capabilities ofthe autonomous driving vehicle related to these algorithms may not becompletely lost.

In examples in which the heterogeneous sensor array 1 is for asurveillance system having more than one sensor type, a surveillancesystem component to ascertain whether to output an alert based on thesensor types (e.g., an intruder alert) may receive the sensor data 19 ordata derived from the sensor data 19 as an input. Surveillance maycontinue to function despite a sensor deficiency of one type of sensor.

Due to the compensation, redundancy for a specific sensor type may notbe required. For example, the heterogeneous sensor array 1 may not needmore than one LIDAR sensor, for instance. However, it may still beadvantageous to use redundancy, and in embodiments using redundancy, theperception engine 15 may generate or identify the sensor data 19 only inthe event of a sensor deficiency affecting one of the sensors 5AN andits redundancy (or redundancies in the case of multiple redundancy).

FIG. 2 illustrates another example system 200 equipped with technologyfor compensating for a sensor deficiency of sensors 25A-N of anautonomous vehicle or a vehicle with an automated driver warning system,according to various embodiments. The vehicle may be a wheeled road oroff-road motor vehicle or any other type of vehicle such as a vehicle tooperate on rails, an aircraft, a watercraft, a robot, or the like, orcombinations thereof.

The sensors 25A-N may generate raw sensor outputs 26A-N, some of whichmay be of different sensor types (e.g., one of raw sensor outputs 26A-N,say raw sensor output 26A, may be for a first sensor type, and anotherone of raw sensor outputs 26A-N, say raw sensor output 26N, may be for asecond different sensor type). In some examples, the first sensor typemay be one LIDAR, camera, radar, or the like, and the other sensor typemay be a different one of LIDAR, camera, radar, or the like. The system200 may include a number of modules to operate on various points ofperception and/or planning pipelines to create an accurate map of thesurroundings, identify obstacles, and plan safe trajectories for thevehicle and/or identify warnings for the driver. These modules mayinclude a multimodal sensor synchronization module 30, obstacledetection modules 35A-N, object fusion module 37, and object predictionmodule 39.

Multimodal sensor synchronization module 30 may synchronize the rawsensor data 26A-N to generate synchronized sensor data 27A-N. Forexample, multimodal sensor synchronization module 30 may performbuffering or other functions to output sensor data 27A-N, which mayinclude synchronized frames for the raw sensor data 26A-N, respectively,in some examples.

In normal operation (e.g., no sensor deficiency), multimodal sensorsynchronization module 30 may receive all of the raw sensor outputs26A-N, and may output the respective sensor data 27A-N to the objectdetection modules 35A-N, respectively. In the normal operation, obstacledetection modules 35A-N may identify obstacle data 36A-N, respectively.The object fusion module 37 may fuse the obstacle data 36A-N into asingle obstacle representation 38 of the surroundings. The objectprediction module 39 may plan a trajectory and/or identify warnings fora driver based on the single obstacle representation 38, and generate acontrol output 40 to one or more vehicle components (such as automaticsteering, automatic braking, a display (visual and/or audio) for awarning, or the like, or combinations thereof) to activate the vehiclecomponents to cause the vehicle to execute safe maneuvers and/or displaywarnings.

In the event of a sensor deficiency, one or more of raw sensor outputs26A-N may be unavailable, and a subset of the sensor data 27A-N may beoutput by the multimodal sensor synchronization module 30. In someembodiments, the sensor translation module 31 may detect one or moremissing frames in one or more modalities from a received input 32 (input32 may include some or all of the same subset of the sensor data 27A-Noutput by the multimodal sensor synchronization module 30 in the eventof the sensor deficiency). Detection may be based on an analysis of thereceived frames of the input 32 and/or a control signal 33 from theaffected ones of the sensors 25A-N or from multimodal sensorsynchronization module 30 to indicate missing frame(s). Upon detection,sensor translation module 31 may provide an output 29, which may includemodeled sensor data for the one(s) of the sensors 25A-N associated withthe sensor deficiency. The output 29 may be provided to one(s) ofobstacle detection modules 35A-35N associated, respectively, with theone(s) of the sensors 25A-N associated with the sensor deficiency. Eachreceiving one(s) of obstacle detection modules 35A-N may continue tooutput its portion of obstacle data 36A-36B based on the modeled sensordata.

As indicated previously, in some embodiments sensor translation module31 may perform translation “on-demand”, e.g., in response to adisruption of the flow of raw sensor data 26A-26N and/or sensor data27A-N. In other embodiments, it may be possible and practical for sensortranslation module 31 to constantly perform translation between variousmodalities (e.g., camera to LIDAR, camera to radar, radar to camera,etc.) and select appropriate translation(s) to be provided to arespective one of the obstacle detection modules 35A-N responsive to adisruption of the flow of raw sensor data 26A-26N and/or sensor data27A-N. In embodiments with translation prior to disruption, zero-shottranslation (the effect where sensor pairs for which there was notdirect training data still produce acceptable translation) may be usedto translate many different sensor combinations (for instances sensorpairs in one-to-one sensor translation) using a sensor translation modelthat may have been trained for only a subset of the sensor combinations.Use of zero-shot translation may reduce a size of sensor translationmodel data to be stored on the vehicle as compared to examples withoutzero-shot translation.

Sensor translation module 31 may be a one-to-one sensor translationmodule (e.g., to provide synthetic camera data based on LIDAR or radar,for example), a one-to-many (e.g., to provide synthetic camera and LIDARdata based on radar, for example), or a many-to-one sensor translationmodule (e.g., to provide synthetic camera data based on LIDAR and radar,for example), or any combination thereof. It may be possible andpractical for a sensor translation module to use more than one sensortranslation model to generate more than one set of modeled sensor datain response to a sensor deficiency. For instance, in response to acamera sensor deficiency, such a sensor translation module may providefirst synthetic camera data based on LIDAR using a first sensortranslation model (LIDAR to camera translation), and second syntheticcamera data based on radar using a second sensor translation model(radar to camera translation). Such an output may be fused with sensorspecific fusion techniques and both synthetic camera data may be fused.

The output 29 may include a confidence score to indicate to the objectfusion module 37 that one of the pipelines is less precise than usualwhich may be taken into account at fusion and prediction time. Also,confidence scoring in the sensor translation model 31 may take place ata known delay that can be accounted for at the obstacle fusion model 37.Modeled sensor data of output 29 may carry the original sensor datatimestamp allowing downstream devices to identify the delay and accountfor that delay.

The output 29 may include more than one translation. For example, themodeled sensor data of output 29 may include first sensor data based ona first subset of the raw sensor data 26A-N and second sensor data basedon a second different subset of the raw sensor data 26A-N (e.g.,synthetic LIDAR sensor data may include first synthetic LIDAR sensordata based on radar and second synthetic LIDAR sensor data based oncamera). One or more of the obstacle detection modules 35A-N may comparethe first and second sensor data prior to generating its obstacle dataof obstacle data 36A-N, and include information based on the comparisonin this portion of the obstacle data 36A-N and/or confidence scoring foreach of the first sensor data and the second sensor data.

FIG. 3 is a flow chart showing a process 300 of compensating for asensor deficiency, according to various embodiments. In block 301, adevice such as the perception engine 15 of FIG. 1 or any other devicedescribed herein, may aggregate perception data from individualperception pipelines, each of which is associated with a respective oneof different types of sensors of a heterogeneous sensor set, to identifya characteristic associated with a space to be monitored.

In block 302, the perception engine may monitor for sensor deficiency.These may be by inspecting the perception data or by evaluating acontrol signal, or combinations thereof. If no sensor deficiency isdetected in diamond 303, then the process 300 may return to block 301.

If a sensor deficiency is detected in diamond 303, then in block 304 theperception engine 15 may, in response to a detection of a sensordeficiency associated with a first sensor of the sensors, derive nextperception data for more than one of the individual perception pipelinesfrom sensor data originating from second sensor(s) of the sensors. Inembodiments utilizing redundancy, the process 300 may bypass block 304if there is a sensor deficiency mitigatable using redundant sensors ofthe same type (if say a redundant sensor of the same type as the sensoraffected by the sensor deficiency is fully operational, then block 304may be bypassed).

Process 300 may be utilized not only with respect to vehicle sensors butto any other multimodal sensor suite that map an overlapping physicalspace. This may include, but is not limited to surveillance and/orrobotics, such as service robotics.

FIG. 4 illustrates a sensor translation model 400, in some embodiments.The sensor translation model 400 may be trained based on neural machinetranslation principles. The sensor translation model 400 may include anencoder network 41, an attention network 49, and a decoder network 50.

The encoder network 41 and decoder network 50 may be formed using an RNN(recurrent neural network) such as a multi layered network composed ofLTSMs (long short-term memory networks). The LTSM layers of the encodernetwork 41 may include a first layer 45, one or more hidden secondlayers 46 (e.g., hidden LSTM layers in a deep LTSM encoder), and a thirdlayer 47. The LTSM layers of the decoder network 50 may include one ormore first layers 52 (e.g., hidden LSTM layers in a deep LTSM decoder),a second layer 53, and a third layer 54.

The encoder network 41 may input data 56 (e.g., initial sensor data) asa feature vector of particular dimensions (sensor resolution), and mayextract the higher level object information in the frame. The decodernetwork 50 may perform the opposite function as the encoder network 41,which may include translating the abstracted frame understanding to asensor data output 59.

The process may be guided (weighted) by an attention network 49, whichmay be composed of a Multilayer Perceptron Network (MPN) that stores thecontext vector of the hidden layers of the encoder network 41. Theattention network 49 may provide the decoder network 50 with weights onspatial regions of interest during the translation of the input data 56(the weights may be part of the bi-directional arrow 51). This may allowthe decoder network 50 to “fix its gaze” on salient objects in the framecontext while translating the targeted sensor output. The decodernetwork 50 may transmit communications to the attention network torequest information from the attention network 49 (these requests may bepart of the bi-directional arrow 51). The sensor data output 59 may beanalyzed using actual sensor data for the sensor to which the sensordata output 59 is to simulate as part of the training process and/orevaluation of the training process.

In some embodiments, the encoder network 41 may extract features fromthe input data 56 and learn the information encoding modelling of theparticular sensor. The first layer 45 of the encoder network 41 mayprovide the feature set extracted from the raw sensor data. In someembodiments, this may be performed using a convolutional neural networkstripped of the classification layer. Other approaches may useconvolutional auto-encoders or variational auto-encoders. The featurevector may then be fed into one or more second hidden layers 46, e.g., aDeep LSTM network. LSTM networks may be particularly suited to “forget”unnecessary information such as noise artifacts picked during thefeature encoding at a particular frame. Because information encoded att_(h) affects the information remembered at t_(h+1) overtime the resultmay be a robust model of information modelling for the particular sensorinput. A third layer 47 (e.g., a softmax layer) may be used to convertthe individual vector scores into a distribution. The output 48 of theencoder 41 at a particular time t_(h) may be the hidden vector h^(enc).

In some embodiments, the attention network 49 may focus the featuretranslation at the decoder network 50 on a subset of the information onthe frame. The general mechanism may work in a similar way that humansight glances around a picture to describe it. At every step of thesensor data translation, the attention network 49 may focus on differentspatial regions of interest. The attention network 49 may focus byproviding a context vector (ẑ_(t)) governed by the following function ø:ẑ_(t) = ø ({h^(enc) _(i)}, {α_(i)}), where h^(enc) _(i) is the featurevector corresponding to the encoder network 41 and α_(i) is the weightof the feature vector. The weight of the feature vector at a particulartime t_(h) is may be computed by an attention model function thatdepends on the previous state t_(h-1). With this model, the attentionstate (region of interest) at a particular time may depend on thefeatures that were translated before.

The function of the decoder network 50 may be to reconstitute the sensordata to the particular target sensor output, in essence followingopposite steps from the encoder network 41. Where the input is thehidden vector from the encoder h^(enc) and the output is the generatedtarget sensor data. In some examples, this process may be performedthrough a deep LSTM network that contains the target sensor model. Thetranslation may be guided by the attention network 49 as describedpreviously. The result of the one or more first layers 52 may be thehidden vector h^(dec) that gets fed into the second layer 53 (e.g., ade-convolution decoder network) trained to output the target sensorformat. The third layer 54 may be similar to the third layer 47.

In some examples, configuration data to represent the sensor translationmodel 400 may be produced using one or more datacenters. Thisconfiguration data may be embedded at manufacture onto a device, such asa vehicle with sensors, a surveillance system with sensors, a roboticscomponent with sensors, or some different types with more than one typeof sensor to monitor an overlapping physical space. In some examples,the configuration data may be downloaded from the datacenter(s) over anetwork to the device after manufacture to provide an update. In someexamples, a processor of the device may re-train the sensor translationmodel 400. Re-training may be used to compensate for differentoperational variables between the ones associated with generation of theinput data and operational variables affecting sensors of the device infield operation (for instance, different weather). Re-training may beconcurrent or non-concurrent with processing perception pipelines. Inconcurrent re-training, an independent processor resources and/ordifferent processing cores of a processor of the device may be used forre-training and perception pipeline processing.

In some examples, after the sensor translation model 400 is trainedand/or retrained, the trained and/or retrained sensor translation modelmay be provided to a processor component (e.g., a processor core) of avehicle with sensors, a surveillance system with sensors, a roboticscomponent with sensors, or some different types with more than one typeof sensor to monitor an overlapping physical space. The provided sensortranslation model may include a descriptor of the topology with theinitialized weights and values in hidden and surface layers that haveresults from training and/or retraining sensor translation on the sensorarray. The training and/or retraining may occur in an independentprocessor component (e.g., a different core of the processor of thevehicle), in some examples. The processing component receiving theprovided sensor translation model may utilize this sensor translationmodel to perform sensor translation tasks at runtime. For instance, thisprocessing component may use this sensor translation model to generatethe confidence scores.

FIG. 5 illustrates an example compute device 500 that may employ theapparatuses and/or methods described herein, according to variousembodiments (for instance, any apparatus and/or method associated withany compute device or electronic device described earlier with respectto FIGS. 1-4 and 6 ). In embodiments, the example compute device 500 maybe installed in an autonomous or semi-autonomous vehicles, i.e.,self-driving vehicles, UAV, robots, and so forth. As shown, the examplecompute device 500 may include a number of components, such as one ormore processors 504 (one shown), at least one communication chip 506,and sensors 507 of different types. The at least one communication chip506 may have an interface to interface with a network to obtain atrained sensor translation model and/or to receive raw sensor data fromadditional remote sensors (not shown).

In various embodiments, the one or more processors 504 each may includeone or more processor cores. In various embodiments, the at least onecommunication chip 506 may be physically and electrically coupled to theone or more processors 504. In further implementations, the at least onecommunication chip 506 may be part of the one or more processors 504. Invarious embodiments, compute device 500 may include printed circuitboard (PCB) 502. For these embodiments, the one or more processors 504and the at least one communication chip 506 may be disposed thereon.

Depending on its applications, compute device 500 may include othercomponents that may or may not be physically and electrically coupled tothe PCB 502. These other components include, but are not limited to, amemory controller (not shown), volatile memory (e.g., dynamic randomaccess memory (DRAM) 520), non-volatile memory such as flash memory 522,hardware accelerator 524, an I/O controller (not shown), a digitalsignal processor (not shown), a crypto processor (not shown), a graphicsprocessor 530, one or more antenna 528, a display (not shown), a touchscreen display 532, a touch screen controller 546, a battery 536, anaudio codec (not shown), a video codec (not shown), a global positioningsystem (GPS) device 540, a compass 542, an accelerometer (not shown), agyroscope (not shown), a speaker 550, and a mass storage device (such ashard disk drive, a solid state drive, compact disk (CD), digitalversatile disk (DVD)) (not shown), and so forth.

In some embodiments, the one or more processor 504, DRAM 520, flashmemory 522, and/or a storage device (not shown) may include associatedfirmware (not shown) storing programming instructions configured toenable compute device 500, in response to execution of the programminginstructions by one or more processor 504, to perform methods describedherein such as compensating for a sensor deficiency in a heterogeneoussensor array. In various embodiments, these aspects may additionally oralternatively be implemented using hardware separate from the one ormore processor 504, flash memory 512, or storage device 511, such ashardware accelerator 524 (which may be a Field Programmable Gate Array(FPGA)).

The at least one communication chip 506 may enable wired and/or wirelesscommunications for the transfer of data to and from the compute device500. The term “wireless” and its derivatives may be used to describecircuits, devices, systems, methods, techniques, communicationschannels, etc., that may communicate data through the use of modulatedelectromagnetic radiation through a non-solid medium. The term does notimply that the associated devices do not contain any wires, although insome embodiments they might not. The at least one communication chip 506may implement any of a number of wireless standards or protocols,including but not limited to IEEE 702.20, Long Term Evolution (LTE), LTEAdvanced (LTE-A), General Packet Radio Service (GPRS), Evolution DataOptimized (Ev-DO), Evolved High Speed Packet Access (HSPA+), EvolvedHigh Speed Downlink Packet Access (HSDPA+), Evolved High Speed UplinkPacket Access (HSUPA+), Global System for Mobile Communications (GSM),Enhanced Data rates for GSM Evolution (EDGE), Code Division MultipleAccess (CDMA), Time Division Multiple Access (TDMA), Digital EnhancedCordless Telecommunications (DECT), Worldwide Interoperability forMicrowave Access (WiMAX), Bluetooth, derivatives thereof, as well as anyother wireless protocols that are designated as 3G, 5G, 5G, and beyond.The at least one communication chip 506 may include a plurality ofcommunication chips 506. For instance, a first communication chip 506may be dedicated to shorter range wireless communications such as Wi-Fiand Bluetooth, and a second communication chip 506 may be dedicated tolonger range wireless communications such as GPS, EDGE, GPRS, CDMA,WiMAX, LTE, Ev-DO, and others.

In various implementations, the compute device 500 may be a component ofa vehicle, a component of a robot, a component of a surveillance system,a laptop, a netbook, a notebook, an ultrabook, a smartphone, a computingtablet, a personal digital assistant (PDA), an ultra-mobile PC, a mobilephone, a desktop computer, a server, a printer, a scanner, a monitor, aset-top box, an entertainment control unit (e.g., a gaming console orautomotive entertainment unit), a digital camera, an appliance, aportable music player, and/or a digital video recorder. In furtherimplementations, the compute device 500 may be any other electronicdevice that processes data.

One or more networks and/or datacenters (similar to any networks and/ordatacenters described herein such as those described with reference toFIG. 4 ) may be used to generate a sensor translation model to be usedby the compute device 500. These networks and/or datacenters may includea system of distributed compute devices that may each include componentssimilar to any of the compute device 500 components. The compute devicesof the networks and/or datacenters may not require sensors 507 as suchcompute devices may receive input sensor data collected from computedevice 500 or some other similar compute device (say a prototype ofcompute device 500) with sensors similar to sensors 507.

Any combination of one or more computer usable or computer readablemedium may be utilized. The computer-usable or computer-readable mediummay be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non- exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer- usable medium may include a propagated data signal withthe computer-usable program code embodied therewith, either in basebandor as part of a carrier wave. The computer usable program code may betransmitted using any appropriate medium, including but not limited towireless, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentdisclosure may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user’s computer,partly on the user’s computer, as a stand-alone software package, partlyon the user’s computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user’s computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider).

Examples

Example 1 is an apparatus for compensating for a sensor deficiency in aheterogeneous sensor array. The apparatus may include a heterogeneoussensor array to monitor a physical space, the heterogeneous sensor arrayincluding a plurality of different types of sensors; and a perceptionengine to: aggregate perception data from individual perceptionpipelines, each of which is associated with a respective one of thesensors, to identify a characteristic associated with the physicalspace; detect a sensor deficiency associated with a first sensor of thesensors; and in response to a detection of the sensor deficiency, derivenext perception data for more than one of the individual perceptionpipelines based on sensor data originating from at least one secondsensor of the sensors.

Example 2 includes the subject matter of example 1 (or any other exampledescribed herein), further comprising the perception engine to,responsive to the detection of the sensor deficiency, access a sensortranslation model to convert the sensor data for the at least one secondsensor into synthetic sensor data for the first sensor, wherein the nextperception data of the perception pipeline of the first sensor is basedon the synthetic sensor data.

Example 3 includes the subject matter of any of examples 1-2 (or anyother example described herein), wherein the sensor translation modelcomprises at least one of a radar-to-LIDAR (light detection and ranging)translation model, a LIDAR-to-camera translation model, or acamera-LIDAR translation model.

Example 4 includes the subject matter of any of examples 1-3 (or anyother example described herein), wherein the at least one second sensorof the sensors comprises a plurality of the sensors, wherein each sensorof the plurality of sensors is different than the first sensor.

Example 5 includes the subject matter of any of examples 1-4 (or anyother example described herein), wherein the perception data comprisesobstacle data, and wherein the physical space comprises a physical spaceassociated with a trajectory, and wherein the characteristic associatedwith the physical space comprises an obstacle characteristic.

Example 6 includes the subject matter of any of examples 1-5 (or anyother example described herein), further comprising the perceptionengine to: responsive to the detection of the sensor deficiency,identify whether a redundant sensor of the same type as the first sensoris available; and access a sensor translation model responsive to theredundant sensor of the same type as the first sensor not available.

Example 7 includes the subject matter of any of examples 1-6 (or anyother example described herein), wherein the sensor deficiency is basedon at least one of sensor inactivity, a sensor malfunction, a sensorfailure, a sensor reset, or security event associated with at least oneof the sensors.

Example 8 includes the subject matter of any of examples 1-7 (or anyother example described herein), wherein the perception engine includesa processor and one or more perception modules operated by theprocessor.

Example 9 includes the subject matter of any of examples 1-8 (or anyother example described herein), wherein the apparatus comprises anautonomous or semi-autonomous mobile device

Example 10 includes the subject matter of any of examples 1-9 (or anyother example described herein), wherein the apparatus comprises avehicle including an automated warning system to display data to adriver based on an identification of the characteristic.

Example 11 is computer readable media for compensating for a sensordeficiency in a heterogeneous sensor array. The computer readable mediamay include executable instructions, wherein the instructions, inresponse to execution by a processor, cause the processor to: aggregateperception data from individual perception pipelines, each of which isassociated with respective one of different types of sensors of aheterogeneous sensor set, to identify a characteristic associated with aspace to be monitored by the heterogeneous sensor set; detect a sensordeficiency associated with a first sensor of the sensors; and inresponse to a detection of the sensor deficiency, derive next perceptiondata for more than one of the individual perception pipelines fromsensor data originating from at least one second sensor of the sensors.

Example 12 includes the subject matter of example 11 (or any otherexample described herein), wherein the perception data comprises atleast one of surveillance data or data obtained from monitoringnavigation of a device along a trajectory.

Example 13 includes the subject matter of any of examples 11-12 (or anyother example described herein), wherein the instructions are further tocause the processor to, responsive to detecting the sensor deficiency,access a sensor translation model to convert the sensor data for the atleast one second sensor into simulated sensor data for the first sensor,wherein the next perception data of the perception pipeline of the firstsensor is based on the simulated sensor data.

Example 14 includes the subject matter of any of examples 11-13 (or anyother example described herein), wherein the sensor translation modelcomprises at least one of a radar-to-LIDAR (light detection and ranging)translation model, a LIDAR-to-camera translation model, or acamera-LIDAR translation model.

Example 15 includes the subject matter of any of examples 11-14 (or anyother example described herein), wherein the instructions are further tocause the processor to: responsive to the detection of the sensordeficiency, identify whether a redundant sensor of the same type as thefirst sensor is available; and access a sensor translation modelresponsive to the redundant sensor of the same type as the first sensornot available.

Example 16 is an apparatus for compensating for a sensor deficiency in aheterogeneous sensor array. The apparatus may include an objectprediction module to output information to be used for at least one oftrajectory planning system or a driver warning system; an object fusionmodule coupled to an input of the object prediction module, the objectfusion module to aggregate perception data from individual perceptionpipelines, each of which is associated with one of a plurality ofdifferent types of sensors of a heterogeneous sensor set, and outputaggregated perception data to the object prediction module; a pluralityof object detection modules coupled to an input of the object fusionmodule, each object detection module to generate information of arespective one of the individual perception pipelines; and a sensortranslation model selectively coupled to at least one input of theplurality of object detection modules, the sensor translation model toconvert information that is based on raw sensor data of a first sensorof the heterogeneous sensor set into synthetic sensor data for a secondsensor of the heterogeneous sensor set in response to a sensordeficiency associated with the second sensor.

Example 17 includes the subject matter of example 16 (or any otherexample described herein), further comprising a multimodal sensorsynchronization module selectively coupled to an input of the sensortranslation model, the multimodal sensor synchronization module togenerate the information based on the raw sensor data.

Example 18 includes the subject matter of any of examples 16-17 (or anyother example described herein), wherein an output of the sensors of theheterogeneous sensor set is coupled to an input of the multimodal sensorsynchronization module.

Example 19 includes the subject matter of any of examples 16-18 (or anyother example described herein), wherein the sensor translation modelcomprises a one-to-many sensor translation model.

Example 20 includes the subject matter of any of examples 16-19 (or anyother example described herein), wherein the sensor translation modelcomprises at least one of a radar-to-LIDAR (light detection and ranging)translation model, a LIDAR-to-camera translation model, or acamera-LIDAR translation model.

Example 21 is a device for compensating for a sensor deficiency in aheterogeneous sensor array. The device may include an automatic brakingand/or steering component; and a heterogeneous sensor array to monitornavigation of the device along a trajectory, the heterogeneous sensorarray including a plurality of different types of sensors; and aperception engine to: aggregate perception data from individualperception pipelines, each of which is associated with a respective oneof the sensors, to identify a characteristic associated with thetrajectory; detect a sensor deficiency associated with a first sensor ofthe sensors; derive next perception data for more than one of theindividual perception pipelines based on sensor data originating from atleast one second sensor of the sensors; and control the automaticbraking and/or steering component based on the next perception data.

Example 22 includes the subject matter of example 21 (or any otherexample described herein), further comprising the processor to,responsive to detecting the sensor deficiency, access a sensortranslation model to convert the sensor data for the at least one secondsensor into synthetic sensor data for the first sensor, wherein the nextperception data of the perception pipeline of the first sensor is basedon the synthetic sensor data.

Example 23 includes the subject matter of any of examples 21-22 (or anyother example described herein), wherein the sensor translation modelcomprises at least one of a radar-to-LIDAR (light detection and ranging)translation model, a LIDAR-to-camera translation model, or acamera-LIDAR translation model.

Example 24 includes the subject matter of any of examples 21-23 (or anyother example described herein), wherein the sensor deficiency is basedon at least one of sensor inactivity, a sensor malfunction, a sensorfailure, a sensor reset, or security event associated with at least oneof the sensors.

Example 25 includes the subject matter of any of examples 21-24 (or anyother example described herein), wherein the sensor translation modelcomprises a one-to-many sensor translation model.

Example 26 is an apparatus for compensating for a sensor deficiency in aheterogeneous sensor array. The apparatus may include means foraggregating perception data from individual perception pipelines, eachof which is associated with respective one of different types of sensorsof a heterogeneous sensor set, to identify a characteristic associatedwith a space to be monitored by the heterogeneous sensor set; and meansfor deriving next perception data for more than one of the individualperception pipelines from sensor data originating from at least onefirst sensor of the sensors in response to a detection of a sensordeficiency associated with a second different sensor of the sensors.

Example 27 includes the subject matter of example 26 (or any otherexample described herein), wherein the perception data comprises atleast one of surveillance data or data obtained from monitoringnavigation of a device along a trajectory.

Example 28 includes the subject matter of any of examples 26-27 (or anyother example described herein), further comprising means fortranslating the sensor data for the at least one second sensor intosensor data for the first sensor responsive to the detection of thesensor deficiency, wherein the next perception data of the perceptionpipeline of the first sensor is based on the sensor data.

Example 29 includes the subject matter of any of examples 26-28 (or anyother example described herein), wherein the means for translatingcomprises at least one of a radar-to-LIDAR (light detection and ranging)translation model, a LIDAR-to-camera translation model, or acamera-LIDAR translation model.

Example 30 includes the subject matter of any of examples 26-29 (or anyother example described herein), further comprising means foridentifying whether a redundant sensor of the same type as the secondsensor is available responsive to the detection of the sensordeficiency; and means for accessing a sensor translation modelresponsive to the redundant sensor of the same type as the second sensornot available.

Example 31 is a method for compensating for a sensor deficiency in aheterogeneous sensor array. The method may include aggregatingperception data from individual perception pipelines, each of which isassociated with respective one of different types of sensors of aheterogeneous sensor set, to identify a characteristic associated with aspace to be monitored by the heterogeneous sensor set; and deriving nextperception data for more than one of the individual perception pipelinesfrom sensor data originating from at least one first sensor of thesensors in response to a detection of a sensor deficiency associatedwith a second different sensor of the sensors.

Example 32 includes the subject matter of example 31 (or any otherexample described herein), wherein the perception data comprises atleast one of surveillance data or data obtained from monitoringnavigation of a device along a trajectory.

Example 33 includes the subject matter of any of examples 31-32 (or anyother example described herein), further comprising translating thesensor data for the at least one second sensor into sensor data for thefirst sensor responsive to the detection of the sensor deficiency,wherein the next perception data of the perception pipeline of the firstsensor is based on the sensor data.

Example 34 includes the subject matter of any of examples 31-33 (or anyother example described herein), wherein the translation utilizes atleast one of a radar-to-LIDAR (light detection and ranging) translationmodel, a LIDAR-to-camera translation model, or a camera-LIDARtranslation model.

Example 35 includes the subject matter of any of examples 31-34 (or anyother example described herein), further comprising: identifying whethera redundant sensor of the same type as the second sensor is availableresponsive to the detection of the sensor deficiency; and accessing asensor translation model responsive to the redundant sensor of the sametype as the second sensor not available.

Although certain embodiments have been illustrated and described hereinfor purposes of description, a wide variety of alternate and/orequivalent embodiments or implementations calculated to achieve the samepurposes may be substituted for the embodiments shown and describedwithout departing from the scope of the present disclosure. Thisapplication is intended to cover any adaptations or variations of theembodiments discussed herein. Therefore, it is manifestly intended thatembodiments described herein be limited only by the claims.

Where the disclosure recites “a” or “a first” element or the equivalentthereof, such disclosure includes one or more such elements, neitherrequiring nor excluding two or more such elements. Further, ordinalindicators (e.g., first, second or third) for identified elements areused to distinguish between the elements, and do not indicate or imply arequired or limited number of such elements, nor do they indicate aparticular position or order of such elements unless otherwisespecifically stated.

1-20. (canceled)
 21. An apparatus, comprising: a heterogeneous sensorarray to monitor a physical space, the heterogeneous sensor arrayincluding a plurality of sensors, wherein at least one sensor of theplurality of sensors is a different type of sensors than another sensorof the plurality of sensors; and perception circuitry connected to theheterogeneous sensor array, wherein the perception circuitry is to: feedsensor data generated by respective sensors of the plurality of sensorsthrough corresponding perception pipelines to detect objects; detect adisruption to first sensor data associated with a first sensor of theplurality of sensors; and in response to the detection of thedisruption, use second sensor data associated with a second sensor ofthe plurality of sensors to compensate for the disruption to the firstsensor data.
 22. The apparatus of claim 21, wherein each of thecorresponding perception pipelines is to: detect one or more objectsbased on the sensor data generated by the respective sensors.
 23. Theapparatus of claim 21, wherein the perception circuitry is to fusedetection data output by each of the corresponding perception pipelinesinto a single representation of a surrounding environment.
 24. Theapparatus of claim 21, wherein the perception pipeline includes a longshort-term memory network (LSTM) model to associate objects acrosssensors and through time.
 25. The apparatus of claim 21, wherein theperception pipeline includes a deep learning model comprising an encodernetwork, an attention network, and a decoder network.
 26. The apparatusof claim 21, wherein the perception pipeline is to, responsive to thedetection of the disruption, access a sensor translation model toconvert the second sensor data into synthetic first sensor data, whereinthe synthetic first sensor data is to compensate for the disruption tothe first sensor data.
 27. The apparatus of claim 26, wherein the sensortranslation model is a deep learning model comprising an encodernetwork, an attention network, and a decoder network.
 28. The apparatusof claim 27, wherein the apparatus includes at least one processor tooperate the corresponding perception pipelines or the sensor translationmodel.
 29. The apparatus of claim 21, wherein the disruption to thefirst sensor data is based on at least one of a missing portion of thefirst sensor data, a corrupted portion of the first sensor data, noisydata among the first sensor data, portions of the first sensor databeing unreliable for object detection, or the first sensor data isdetermined to be unusable for perception.
 30. The apparatus of claim 21,wherein the disruption to the first sensor data is based on at least oneof inactivity of the first sensor, a malfunction of the first sensor,failure of the first sensor, a reset of the first sensor, or securityevent associated with the first sensor.
 31. The apparatus of claim 21,wherein the apparatus is disposed in an autonomous or semi-autonomousdriving vehicle, a robot, or a mobile device.
 32. One or morenon-transitory computer readable media (NTCRM) comprising instructions,wherein execution of the instructions by one or more processors of acomputing device is to cause the computing device to: facilitate sensordata generated by respective sensors of a plurality of sensors to flowto corresponding object detectors, wherein at least one sensor of theplurality of sensors is a different type of sensors than at least oneother sensor of the plurality of sensors; detect a disruption to firstsensor data associated with a first sensor of the plurality of sensors;and in response to the detection of the disruption, use second sensordata associated with a second sensor of the plurality of sensors tocompensate for the disruption to the first sensor data.
 33. The one ormore NTCRM of claim 32, wherein execution of the instructions is tocause the computing device to operate each of the corresponding objectdetectors to: detect one or more objects based on the sensor datagenerated by the respective sensors.
 34. The one or more NTCRM of claim32, wherein execution of the instructions is to cause the computingdevice to operate each of the corresponding object detection modules to:perform data fusion on detection data output by each of thecorresponding object detectors into a single representation of asurrounding environment and through time.
 35. The one or more NTCRM ofclaim 32, wherein the disruption to the first sensor data is based on atleast one of a missing portion of the first sensor data, a corruptedportion of the first sensor data, noisy data among the first sensordata, portions of the first sensor data being unreliable for objectdetection, or the first sensor data is determined to be unusable forperception, inactivity of the first sensor, a malfunction of the firstsensor, failure of the first sensor, a reset of the first sensor, orsecurity event associated with the first sensor.
 36. The one or moreNTCRM of claim 32, wherein execution of the instructions is to cause thecomputing device to: operate a long short-term memory network (LSTM)model to associate objects across sensors and through time.
 37. The oneor more NTCRM of claim 32, wherein execution of the instructions is tocause the computing device to: operate a sensor translation model toconvert the second sensor data into synthetic first sensor data, whereinthe synthetic first sensor data is to compensate for the disruption tothe first sensor data.
 38. The a one or more NTCRM of claim 37, whereinthe sensor translation model is a deep learning model comprising anencoder network, an attention network, and a decoder network.
 39. Acomputing system, comprising: interface circuitry to connect thecomputing system to one or more mechanical components, and connect thecomputing system to a heterogeneous sensor array to monitor a physicalspace surrounding the autonomous or semi-autonomous system, wherein theheterogeneous sensor array includes a plurality of sensors, wherein atleast one sensor of the plurality of sensors is a different type ofsensors than other sensors of the plurality of sensors; and perceptioncircuitry connected to the heterogeneous sensor array via the interfacecircuitry, wherein the perception circuitry is to: feed sensor datagenerated by respective sensors of the plurality of sensors throughcorresponding perception pipelines to corresponding object detectors;detect a disruption to first sensor data associated with a first sensorof the plurality of sensors; in response to the detection of thedisruption, use second sensor data associated with a second sensor ofthe plurality of sensors to compensate for the disruption to the firstsensor data; determine a trajectory based on perception data output bythe corresponding object detectors, wherein the perception data is basedon the compensation for the disrupted first sensor data; and control theone or more mechanical components based on the determined trajectory.40. The computing system of claim 39, wherein the one or more mechanicalcomponents include one or more of a propulsion system, a steeringmechanism, and a braking mechanism.
 41. The computing system of claim39, wherein each of the corresponding object detectors is to: detect oneor more objects based on the sensor data generated by the respectivesensors.
 42. The computing system of claim 39, wherein the perceptioncircuitry is to feed the perception data output by each of thecorresponding object detectors to a data fusion module to fuse theperception data into a single representation of a surroundingenvironment and through time.
 43. The computing system of claim 39,wherein the perception circuitry includes a long short-term memorynetwork (LSTM) model to associate objects across sensors and throughtime.
 44. The computing system of claim 39, wherein the perceptioncircuitry is to operate a deep learning model, wherein the deep learningmodel includes an encoder network, an attention network, and a decodernetwork.
 45. The computing system of claim 39, wherein the deep learningmodel is a sensor translation model, and the perception circuitry is tooperate the sensor translation model to convert the second sensor datainto synthetic first sensor data, wherein the synthetic first sensordata is to compensate for the disruption to the first sensor data.